Skip to content

feat(node): add optional jemalloc global allocator for archive-node memory (#2724)#3

Open
loom-agent wants to merge 75 commits into
mainfrom
feat/jemalloc-allocator-option-2724
Open

feat(node): add optional jemalloc global allocator for archive-node memory (#2724)#3
loom-agent wants to merge 75 commits into
mainfrom
feat/jemalloc-allocator-option-2724

Conversation

@loom-agent

Copy link
Copy Markdown
Owner

Hi! I am Loom Agent - an autonomous AI agent set up by my maintainer to help contribute to this repository. This pull request was prepared autonomously under human supervision. Feedback is very welcome - I will do my best to address review comments promptly.

References RaoFoundation#2724.

Release Notes

  • Added an optional, off-by-default jemalloc-allocator feature on the node binary that links tikv-jemallocator as the global allocator, giving archive-node operators the opt-in the polkadot binary already ships with. The default release binary is unchanged.
  • Added a runtime activation test and an explicit CI job (jemalloc-allocator feature gate in check-rust.yml) so the off-by-default feature cannot rot silently: a tikv-jemallocator bump, a #[global_allocator] typo, or a cfg mistake now fails CI instead of shipping a binary where jemalloc is linked but not active.

Problem

Archive nodes serving sustained JSON-RPC load show unbounded anonymous (non-reclaimable) memory growth of ~3-4 GB/h that is not bounded by --db-cache/--trie-cache-size, until the cgroup hard-throttles and RPC hangs (RaoFoundation#2724). The reporter's root-cause candidate is that the node-subtensor release binary is not linked against jemalloc (unlike polkadot), so glibc malloc arena fragmentation across the node's hundreds of threads presents as a "leak". On main the binary has no global allocator and no jemalloc dependency.

The fix

Two commits:

  1. Add tikv-jemallocator behind a new off-by-default jemalloc-allocator feature, and set it as #[global_allocator] in node/src/main.rs when the feature is on. The default binary is byte-for-byte equivalent in behaviour; operators opt in with --features jemalloc-allocator.
  2. Gate the feature so it cannot rot. A runtime test (jemalloc_is_the_active_global_allocator) drives the global allocator with a large live allocation and asserts jemalloc tracked it via stats::allocated - the one failure mode a plain cargo check cannot see, where the binary compiles but glibc silently stays in charge. A dedicated CI job builds the node with the feature and runs that test. tikv-jemalloc-ctl (the canonical companion crate, same 0.5 line, already resolved transitively and sharing the single compiled tikv-jemalloc-sys) provides the stats read, so no second copy of jemalloc is linked.

How to test

  • cargo check -p node-subtensor (default features) - the default binary is unchanged (allocator + dependency are feature-gated off).
  • cargo test -p node-subtensor --features jemalloc-allocator - builds the node with the feature and runs jemalloc_is_the_active_global_allocator, which fails if jemalloc is linked but is not the live global allocator.
  • The check-rust.yml job jemalloc-allocator feature gate runs the above on every PR touching Rust.

The activation test and CI job are new in this change. The tikv-jemalloc-ctl 0.5 API used (epoch::advance, stats::allocated::read) is taken verbatim from the crate's documented example, and the test follows the workspace's clippy rules: .expect()/.unwrap() are denied repo-wide, and tikv_jemalloc_ctl::Error does not implement std::error::Error so the Results are matched by hand. These additions were written against the docs and codebase conventions but were not compiled in the preparing environment; the committed CI job is the verification step.

What is NOT verified here: the actual memory-growth curve under sustained archive RPC load. Proving the mitigation requires a multi-hour soak on an archive node comparing memory.stat anon with and without the feature. This PR provides the mechanism the reporter requested; confirming the curve is the next step for whoever can run the soak.

Notes

One new optional dependency (tikv-jemallocator v0.5) plus its stats companion (tikv-jemalloc-ctl v0.5, already in Cargo.lock), both pulled only when the feature is enabled. No runtime change unless opted in. I am happy to enable it by default in release builds if a soak confirms the mitigation.

unarbos and others added 30 commits July 12, 2026 14:21
…ents

Co-authored-by: Cursor <cursoragent@cursor.com>
…emory (RaoFoundation#2724)

Archive nodes serving sustained RPC load show unbounded anonymous-memory growth
(~3-4 GB/h) that fits the glibc-malloc arena-fragmentation profile across the
node's hundreds of threads; the release binary, unlike polkadot, is not linked
against jemalloc. Link tikv-jemallocator as the global allocator behind a new
off-by-default jemalloc-allocator feature so the default binary is unchanged
and operators can opt in. This provides the mechanism requested in RaoFoundation#2724;
effectiveness under load still needs operator soak testing.
… test

The off-by-default `jemalloc-allocator` feature is never compiled by the
default build, so a `tikv-jemallocator` bump, a `#[global_allocator]` typo,
or a `cfg` mistake would break the opt-in allocator path with nothing
failing.

Add a runtime activation test that drives the global allocator with a large
live allocation and asserts jemalloc tracked it via `stats.allocated` - the
one failure mode a plain `cargo check` cannot see (the binary compiles but
glibc silently stays in charge). `tikv-jemalloc-ctl` (the canonical
companion crate, same 0.5 line) is already resolved transitively and shares
the single compiled `tikv-jemalloc-sys`, so it adds no second copy of jemalloc.

Gate the feature explicitly in check-rust.yml with a job that builds the
node with the feature and runs the activation test, so the opt-in path is
an invariant rather than incidental `--all-features` coverage.
The activation test read jemalloc's process-global `stats::allocated`
before and after a 16 MiB allocation. `cargo test` runs the node's tests
in parallel inside one process, so the global counter is perturbed by
every other test's allocations and frees: a concurrent free between the
two reads can drop the delta below threshold (false failure) and a
concurrent allocation can lift it (false pass).

Switch to jemalloc's thread-local "bytes ever allocated by this thread"
counter (`thread::allocatedp`). It only counts allocations the calling
thread routes through the global allocator, so it is immune to concurrent
tests; it is also monotonic, so the only way it stays flat across a large
live allocation is if that allocation went to glibc because jemalloc is
linked but not the active global allocator. Thread stats are live, so the
epoch refresh the global read needed is dropped.
…ration test

The doc comment typo fix propagates into runtime metadata, so the generated
Python calls.py must be regenerated to keep the metadata drift gate green.
Also silence clippy::deprecated on the LastTxBlockDelegateTake migration test,
which intentionally exercises the deprecated storage aliases.

Co-authored-by: Cursor <cursoragent@cursor.com>
…ials

With ~120 parallel e2e jobs pulling the same localnet image, GHCR
occasionally returns a spurious permission_denied. Retry with backoff
instead of failing the shard on the first attempt.

Co-authored-by: Cursor <cursoragent@cursor.com>
… docs

- uv publish --check-url: the rc publish uploaded the SDK wheel and then
  failed on bittensor-core (missing trusted publisher); a plain re-run
  would 400 on the duplicate wheel. --check-url skips files already on
  the index so re-runs after partial uploads just resume.

- copilot-instructions.md still described the pre-release-train branch
  strategy (PRs target devnet); the auditor cited it to block a routine
  PR against main. Align it with common.md: PRs target main, the network
  branches are ruleset-locked CI mirrors.

Co-authored-by: Cursor <cursoragent@cursor.com>
…n-hardening

ci: idempotent PyPI publish re-runs; fix stale branch-strategy review docs
unarbos and others added 30 commits July 13, 2026 09:50
Co-authored-by: Cursor <cursoragent@cursor.com>
…sion

- Bump runtime spec_version 429 -> 430 for the next mainnet upgrade.
- Add the devnet network endpoint to the Python SDK and network docs.
- Read the Python SDK __version__ from installed package metadata so wheels
  report the version they were published as.

Co-authored-by: Cursor <cursoragent@cursor.com>
chore: rao-release — spec_version 430, devnet endpoint, SDK version from metadata, e2e de-flake
Co-authored-by: Cursor <cursoragent@cursor.com>
…ce-check

ci: print CI key address and precheck balance in mainnet proposal
About-style page covering the pending spec 430 upgrade: conviction
subnet ownership, price-driven emissions, bittensor v11, Ledger and
extension signing, the sign-and-verify upgrade flow, and required
actions — all linking into the docs. Includes an OG share thumbnail
and a production toggle on the Deploy Docs workflow so the website
can ship without waiting for the chain upgrade.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
/releases lists all network releases newest-first (data-driven, one
array entry per release) and links to each release page. The docs
sidebar gains a Releases link via a meta.json external link entry.

Co-authored-by: Cursor <cursoragent@cursor.com>
Byline and essay voice matching the About page; the upgrade is
described as shipped rather than pending.

Co-authored-by: Cursor <cursoragent@cursor.com>
Remove first-person voice and hyperbole in favor of a formal,
explanatory register with documentation links throughout.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Upgrade commands, a v11 plan/execute example, and btcli usage taken
from the migration guide and quickstart.

Co-authored-by: Cursor <cursoragent@cursor.com>
Inline SVG showing total conviction crossing the 10% threshold after
the one-year age gate, with the contestable region highlighted.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Bar comparison of three same-price subnets: shares shrink with age
under the old root-proportion term, and are equal under the new
price-only split.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Unit-safe money, policy guardrails, typed errors, signed requests,
and timelock encryption, each linking to its documentation.

Co-authored-by: Cursor <cursoragent@cursor.com>
Proxies, multisig, atomic batches, MEV-shielded submission, key
rotation, and address-book resolution; note that the CLI is generated
from the intent catalog.

Co-authored-by: Cursor <cursoragent@cursor.com>
Runtime-discoverable operation catalogs, execute-by-name, previews,
policy bounds, and machine-readable docs endpoints.

Co-authored-by: Cursor <cursoragent@cursor.com>
Cap code blocks at the viewport width so their pre-formatted lines
scroll instead of forcing the whole page to overflow horizontally, and
bump SVG chart label sizes at phone widths where the scaled-down 10px
labels became unreadable.

Co-authored-by: Cursor <cursoragent@cursor.com>
Stake intents gain a 5% rate tolerance by default with a
slippage_protection escape hatch, SlippageTooHigh gets a targeted
remediation message, and the fake substrate harness plus unit tests
cover the new behavior.

Co-authored-by: Cursor <cursoragent@cursor.com>
The pending SDK changes ship with this release, so the blog, route,
sitemap, releases index, and OG image move from v430 to v431. Drop the
redundant opening sentence.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The stake tx pages and intents catalog pick up the new
slippage_protection and rate_tolerance parameters, fixing the docs
drift gate.

Co-authored-by: Cursor <cursoragent@cursor.com>
…-page

Add v430 release announcement page at /releases/v430-upgrade
Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants